Smoothing

Signal and Noise

• Data scientists tend to view the trends in a dataset as being influenced by two components:
  • Signal - the true underlying pattern or relationship
  • Noise - unexplainable variability that obscures the signal
• This framework is very similar the assumptions made by familar statistical models (like linear regression or the \(t\)-test)

Smoothing

• Upon acknowledging that our data has noise, simple visualizations might not be effective in highlighting the true patterns or relationships we hope to discover (ie: the signal)
  • Smoothing refers to the process of using a function to capture signal while excluding noise
  • Linear regression is a type of smoothing function, it enforces a straight-line relationship between \(X\) and \(Y\), deeming any deviation from that line to be noise

Regression in Two-dimensions

• Multiple linear regression can extend the idea of smoothing into two-dimensions, expressing the signal as a plane (defined by separate slopes in the \(x_1\) and \(x_2\) dimensions)

LOESS

• Linear regression is a very inflexible smoother, it cannot capture non-linear signals
• LOESS, short for locally estimated scatterplot smoothing, is a generalization of moving averages
  • For every datapoint, LOESS defines a neighborhood and fits a weighted regression line on data-points within that neighborhood
  • These local regression lines are pieced together to form a smooth trendline

Animation from “Introduction to Data Science - Data Analysis and Prediction Algorithms with R” by Rafael A. Irizarry

LOESS

• Choosing an appropriate neighborhood is crucial in order for LOESS to provide a good approximation of the signal in a dataset
• Neighborhood size is controlled by a parameter known as span
  • When span = 0.5, the nearest 50% of the data will be used to fit the local regression at the point \(x_0\)
  • So span = 0.5 and \(n = 100\), the 50 datapoints closest to \(x_0\) will be included in the neighborhood
  • The animation below illustrates different span specifications

Animation from “Introduction to Data Science - Data Analysis and Prediction Algorithms with R” by Rafael A. Irizarry

LOESS

• Returning to our toy example, using LOESS with span = 0.2 is still too noisy, but increasing to span = 0.6 does a good job capturing the signal

p1 <- ggplot(df, aes(x = x, y = y)) + geom_line(color = "red") + labs(title = "Signal")

p2 <- ggplot(df, aes(x = x, y = yy)) + geom_point() + geom_line(aes(x = df$x, y = df$y), color = "red") + geom_smooth(method = "loess", span = 0.2) + labs(title = "span = 0.2")

p3 <- ggplot(df, aes(x = x, y = yy)) + geom_point() + geom_line(aes(x = df$x, y = df$y), color = "red") + geom_smooth(method = "loess", span = 0.5) + labs(title = "span = 0.6")

grid.arrange(p1,p2,p3)

Splines

• LOESS works well on small to moderate datasets, but for large datasets it is too computationally intensive
• An alternative is to use splines, which are continuous functions formed by joining polynomial approximations at locations called “knots”
• The mathematical details of splines are beyond the scope of this course, but you should know that they can be used to smooth large datasets, though they sometimes perform worse than LOESS on smaller datasets

Takeaways

• Smoothing is a tool to help identify trends in noisy data
• Smoothing isn’t always necessary, sometimes the trends are obvious
  • In the example on the left the strong linear relationship between a home’s assessed value and it’s sale price is obvious without the smoother
  • In the example on the right it would be difficult to notice the increase in the number of bedrooms for homes built in the 1960s without the smoother